8 Parallel File Systems

The success of a CDI Grid is dependent upon the design of its storage infrastructure. As seen in Chapter 7, processing in this environment revolves around the simultaneous movement and transformation of data on many compute elements. Effective storage solutions combine hardware and software to meet these needs. The storage hardware selected must provide enough raw throughput for the expected workloads. Typical storage hardware architectures also often provide some redundancy to help in creating a fault tolerant system. Storage software, specifically file systems, must organize this storage hardware into a single logical space, provide efficient mechanisms for accessing that space, and hide common hardware failures from compute elements. Parallel file systems (PFSes) are a particular class of file systems that are well suited to this role. This chapter will describe a variety of PFS architectures, but the key feature that classifies all of them as parallel file systems is their ability to support true parallel I/O. Parallel I/O in this context means that many compute elements can read from or write to the same files concurrently without significant performance degradation and without data corruption. This is the critical element that differentiates PFSes from more traditional network file systems, and it is this characteristic that makes PFSes an integral part of an effective CDI Grid. While databases play a significant role in data engineering for the grid, they are not appropriate for all types of data. PFSes are particularly well-suited to storing large amounts of structured data that can reside within flat files. This type of data can be dynamically distributed and processed across almost any number of compute elements by simply assigning each compute element a unique portion of the file. Compute elements can then be added incrementally to increase the processing rate. This corresponds well with CDI processes that will be discussed in greater